Add Qwen3-1.7B iOS and macOS export support - #196

Move the duplicated private generateNoise(count:seed:) from
Flux2Pipeline, SD3Pipeline, and StableDiffusionPipeline into a
module-level free function in RNG/NoiseGeneration.swift.

The new function accepts a RandomSourceType parameter (.numPy,
.torch, .nvidia) defaulting to .numPy for backwards compatibility,
allowing callers to select the appropriate RNG for their model.

* Extract lastSafeIndex into shared free function

* Add unit tests for lastSafeIndex streaming marker holdback

* Additional unit tests.

…ple#177)

Move findImageInputName, findLogitsOutputName, and findBoxesOutputName
into a new ModelIONameResolver enum in CoreAIShared/Runtime, eliminating
duplication between ObjectDetector and ImageSegmentationEngine.

… (apple#181)

findImageInputName and findLogitsOutputName moved to
ModelIONameResolver in CoreAIShared — update test references.

* Add Muse Glimmer 30B text decoder support

Meta's on-device agentic model (Apache 2.0). Architecture features:
- Local/Global attention: [S,S,S,G] repeating (39+13 layers)
- CenteredRMSNorm (1+weight) for layer norms, plain RMSNorm for final norm
- Weight-less RMSNorm on embeddings
- QK norm (shared RMSNorm on Q/K per-head) + qk_scale_factor on Q
- Gated attention: sigmoid(gate_proj(x)) * attn_output
- Extreme GQA: 32Q / 2KV (16:1 ratio)
- Per-layer RoPE control (global layers skip RoPE)
- output_multiplier (0.196) and logit softcapping (20.0)

Evaluated: word_ppl = 7.71 (FP16), ~8.4 (INT4).

…e#185)

* Extract LogProbabilities helper with Accelerate-vectorized log-softmax

Move log-probability computation out of ContinuationEvaluationResult into
a dedicated LogProbabilities struct. The new implementation uses vDSP for
Float16→Float32 conversion, max, subtract, and sum operations (10-30×
faster than scalar loops for typical vocabulary sizes).

Handles edge cases:
- +Inf logits: dominant token gets log-prob 0, others get -Inf
- Invalid token indices: -Inf log-prob, skipped in sum/mean
- Very large logits: numerically stable via max-subtraction

ContinuationEvaluationResult now delegates to LogProbabilities.compute()
instead of reimplementing log-softmax inline.

17 unit tests covering correctness, edge cases, and numerical stability.

* Apply suggestions from code review

Co-authored-by: Alejandro Isaza <167236+alejandro-isaza@users.noreply.github.com>

* Use vImageConvert for Float16→Float32, stack-allocate temporaries

- Float16→Float32 conversion via vImageConvert_Planar16FtoPlanarF
- shiftedBuffer and expBuffer use withUnsafeTemporaryAllocation
- Remove unused infCount variable

---------

Co-authored-by: Alejandro Isaza <167236+alejandro-isaza@users.noreply.github.com>

* Add repetition penalty support for CPU-based engines

Penalizes tokens that appear in recent generation history, discouraging
repetitive output. Applied as a separate logit modification step before
the existing sampling pipeline (temperature/topK/topP/minP).

- Add repetitionPenalty and repetitionPenaltyWindow to SamplingConfiguration
- Add RepetitionPenaltyProcessor (deduplicates, sign-aware divide/multiply)
- Integrate into Sequential, StaticShape, VLM, and Constrained engines
- Only penalize generated tokens (not prompt) via generationStartOffset
- Pipelined engine: hard fail with clear error (GPU path in follow-up)
- CLI: --repetition-penalty and --repetition-penalty-window flags

* Add GPU repetition penalty for pipelined engine

Extend MPSGraphCompositeSampler with an optional penalty stage (penaltyEnabled
flag at init). When active, the compiled graph applies sign-aware penalty
(divide positive logits, multiply negative) before topK.

Refactor the monolithic graph-building init into composable static stage
helpers (applyPenaltyStage, topKStage, temperatureStage, softmaxStage,
minPStage, topPStage, maskAndNormalizeStage, multinomialStage, gatherTokenStage)
that can be unit-tested independently.

RepetitionPenaltyGPUState manages per-pipeline-depth rotating penalty buffers
with dirty-tracking: recordToken() updates only CPU-side ring state, and
buffer(forStep:) applies pending writes at encode time when the gate guarantees
no in-flight GPU read on that slot.

Inherent 2-token staleness from pipelineDepth=3 is acceptable for practical
window sizes. Greedy + penalty on pipelined is rejected at entry (use sequential).

* Fix validation, silent drops, and force-unwraps in repetition penalty

- validate(): reject penalty < 1.0, orphan window, and penalty + json-schema
  (constrained generation does not support penalty on the pipelined engine)
- Greedy strategy: forward repetition penalty to SamplingConfiguration
  (was silently constructing config without it)
- Force-unwraps: replace repetitionPenalty! with guard-let in
  ConstrainedDecodingStrategy and ConstrainedGenerator
- fallbackSampler(from:): precondition catches wrong overload usage

* Add sentinel test for MPSGraph completion ordering assumption

RepetitionPenaltyGPUState relies on completions firing in submission
order (no additional synchronization). This is observed behavior on a
single MTLCommandQueue but not documented by Apple. The test validates
the assumption and will break if the dispatch model changes.

* Propagate encode errors via completion instead of swallowing with try?

Also note GPU window cap (256) in --help text.

* Fix swift-format lint warnings

* Use feedTensors to order runAsync inputs for penalty encode

The feeds dictionary used at compile time has no guaranteed order.
Using executable.feedTensors ensures the inputs array matches the
order the compiled graph expects.

* Add Phi-3/3.5/4-mini-instruct support

Microsoft's Phi family (MIT license, 3.8B). Architecture features:
- Fused QKV projection (MHA models) and fused gate_up MLP
- LongRoPE for 131K context (Phi-3.5, Phi-4)
- Sliding window attention (Phi-3, window=2047)
- GQA 24Q/8KV (Phi-4) and MHA 32/32 (Phi-3, 3.5)
- INT4 quantization with embedding excluded (tied weights)

Perplexity (wikitext-2): Phi-3 9.47, Phi-3.5 9.98, Phi-4 11.12 (FP16)
Within 0.3% of HuggingFace transformers baseline.

* Trigger CI re-run

* Fix test: set rope_parameters for transformers 5.12+ compatibility

Phi3RotaryEmbedding in transformers 5.12.1 requires rope_parameters
to be a dict (not None). Set {"rope_type": "default"} on test configs.

* Update eval table: show custom YAML compression (embedding excluded)

Per review: clarify that INT4 uses phi_4bit_embedding_excluded.yaml
(not the default 4bit preset). Add BPW footnote explaining why.

* Fix test: add rope_theta to rope_parameters dict

transformers 5.12.1 Phi3RotaryEmbedding reads rope_theta from
config.rope_parameters["rope_theta"], not config.rope_theta.

* Loosen multi-token parity tolerance for transformers 5.12+

HF 5.12.1 changed Phi3RotaryEmbedding to use rope_init_fn which
computes slightly different frequencies than our LongRoPE/initialize_rope.
Max diff is 0.0094 (well within correctness — PPL validates within 0.3%).
Relax multi-token test to atol=1e-2 to accommodate cross-implementation
RoPE numeric differences.

* Support agentic chain-of-thought format in ThinkTagParser

Extend ThinkTagParser to handle models that use to=self/to=user
message routing for chain-of-thought, in addition to the existing
symmetric tag-pair format (<think>/</think>).

ThinkTagParser.Format enum:
  .tagPair(open:close:) — existing behavior, unchanged
  .agentic(selfMarker:userMarker:endOfMessage:endOfTurn:)

detectThinkingFormat probes the tokenizer vocab:
  - <|eom|> + <|eot|> + <|message|> → .agentic
  - <think>/<think> or <|reasoning_start|>/<|reasoning_end|> → .tagPair
  - fallback → .tagPair with default markers

For agentic models, <|eot|> is added to the EOS set so generation
stops after the first user-facing response.

25 unit tests.

* Fix swift-format: import order and line length in tests

* Fix token-by-token streaming in agentic parser

Strip entry markers at the top of each loop iteration. When streaming
char-by-char, after <|eom|> is consumed the buffer is empty — the
following entry marker hasn't arrived yet. The inline hasPrefix check
after the transition finds nothing, so the marker leaks as text on
subsequent consume() calls.

)

Co-authored-by: Tao Jia <tjia1818@users.noreply.github.com>

This commit introduces a streaming mode for Parakeet to enable live transcription via buffered/chunked inference. Offline transcription Is largely unaffected and retains the same behavior/accuracy as before, with some additive, non-breaking public API changes.

The primary use case for Parakeet streaming is through the Swift APIs for app integration. Optionally, the CLI exposes the streaming interface for diagnostic and correctness testing purposes.

NOTE: Streaming is for streaming bundles ONLY. Static and dynamic bundles will be rejected.

* Add OpenAI-compatible server mode to llm-runner

Adds `llm-runner serve` — an HTTP server exposing OpenAI-compatible
endpoints for on-device LLM inference.

Endpoints:
  POST /v1/chat/completions  — streaming + non-streaming
  POST /v1/completions       — per-token logprobs (sequential engine)
  GET  /v1/models            — model list
  GET  /health               — liveness probe

Also includes:
- Batched forcedContinuation for faster perplexity evaluation
- ThinkTagParser.stripCompleted for --no-thinking mode
- --raw-tokens support for pre-tokenized input

Hummingbird v2 for HTTP. Single-request concurrency (429 for overlap).
Logprobs require sequential engine variant.

* Split llm-server into separate target with CoreAILMCommon library

Separate the OpenAI-compatible server into its own executable target so that
llm-runner no longer depends on Hummingbird. Shared API types live in
CoreAILMCommon — an internal library independently testable by both tools.

- CoreAILMCommon: ServerAPITypes + CompletionTypes (pure Codable structs)
- llm-server: Hummingbird HTTP server, chat/completion handlers, server state
- llm-runner: loses server/ directory and Hummingbird dependency
- ThinkTagParser.stripCompleted: rewritten as single-pass O(n)
- Non-streaming handler: deduplicated strategy selection
- CoreAILMCommonTests: 30 tests covering all wire-format types

* Address review: Prompt enum, throw on buffer underflow, padding comment

- Replace __TOKEN_IDS__ magic string with typed Prompt enum (.text/.tokenIds)
  in CoreAILMCommon — single source of truth for prompt format
- CoreAISequentialEngine: throw on batched logits buffer underflow instead
  of silently breaking (would leave inconsistent count state)
- CompletionHandler: add comment explaining padding token purpose for
  lm-eval's [ctxlen:-1] slicing convention
- Update tests to use new Prompt enum assertions

* Fix timing display, return 400 for decode errors, preserve no-thinking in fallback

- Add Task.yield() after modelLoadSpan.end() so StatsStorage recording
  task runs before reading modelLoadTime (fixes 0.000s display)
- Return HTTP 400 (not 500) for malformed request bodies (DecodingError)
- Use templateMessages (with /no_think annotations) in tokenizer fallback
  path so thinking suppression works even without chat template support

* Fix streaming think-tag corruption: use ThinkTagParser.consume() instead of stripCompleted

The streaming path was reapplying stripCompleted to the full accumulated
text on every token, computing deltas via a high-water mark. If a <think>
tag arrived across token boundaries (e.g., token1="Hello<", token2="think>"),
the "<" was sent to the client before the tag was recognized. When later
stripped, the delta computation lost characters.

Replace with the streaming ThinkTagParser which holds back characters that
could form a partial tag, only emitting .text events that are safe to send.
This is exactly what the parser was designed for.

Also make ThinkTagParser.Event, init, consume(), and flush() public so the
server target can use the streaming protocol.

* Add streaming stats recording, use SuspendingClock for all server timing

- Streaming path now calls state.stats.record() so periodic summary
  includes streaming request throughput
- Replace ContinuousClock with SuspendingClock across all handlers
  (doesn't count time when process is suspended)

* Cap logprobs at 20, use Int32(exactly:) for token ID validation

- Clamp logprobs parameter to max 20 (matches OpenAI limit), prevents
  resource exhaustion from vocab-sized sort+decode per token position
- Replace trapping Int32($0) with Int32(exactly:) that returns a proper
  DecodingError for out-of-range token IDs instead of crashing the server

The static-shift path (SD3 with shift=3.0) computed sigmas incorrectly:

  Old: linspace(1.0, 1/stepCount, N) then shift
  New: linspace(sigma_max*T, sigma_min*T, N) / T then shift

This produced wrong noise levels — the old schedule wasted the last
step with a huge jump (0.429->0.0) instead of smooth denoising (0.009->0.0).

Verified with SD3.5-Medium: the new schedule converges slightly faster
and produces fewer broken artifacts, especially at low step counts.

The dynamic-shift path (Flux with mu) is unchanged.

Also removes local_files_only=True from the diffusion export pipeline
which blocked first-time exports when HF cache was empty.

Wan 2.1 T2V 1.3B generates 480p video (up to 81 frames / 5 seconds)
from text prompts using a 3D DiT transformer with flow matching.

Components:
- CoreAIVideoDiffusionPipeline: Swift video generation pipeline
  (text encode -> denoise -> VAE decode -> frame assembly -> MP4)
- videodiffusion-runner: CLI tool with quality presets (fast/balanced/best)
- Python export: wan.py wrapper + diffusion pipeline integration

Features:
- Sequential CFG with dynamic cutoff (--cfg-cutoff) for speed
- Tiled VAE decode for memory-constrained devices (--tile-size)
- Full temporal VAE decode with padding for shorter clips
- Quality presets with validated step/cfg-cutoff combinations
- INT4/INT8 quantization support via --compression flag
- Dynamic temporal dimension (supports 17-81 output frames)

---------

Signed-off-by: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com>

- Add wan-t2v-1.3b to SUPPORTED_MODELS in models.py (enables short name in CLI)
- Add wan-t2v-1.3b preset to model_registry.py (enables --list-models)
- Revert uv.lock to public PyPI sources

* remove pre-computed RoPE from Flux2 transformer and instead compute in-graph. Stride aware NDArray helpers to accomodate this change. Small metadata write fix.

* uv lock fixes